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Got  a  Wicked  Problem? 


Irregular  Warfare  (IW)  analysis 

is  a  “Wicked  Problem” 


-  IW:  Battlegrounds  of  social 
concepts 

•  Legitimacy 

•  Popular  Will 

-  Many  perspectives 

-  Seems  unsolvable 

•  Two  complementary  approaches 
to  analysis: 

-  Human:  Wargaming 

-  Machine:  Simulation 

•  The  Oz  Wargame  Integration 
Toolkit 

-  A  solution  that  takes  the  best 
of  both  approaches 

-  Integrates  wargames, 
simulations,  rule-based 
systems,  and  data 
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Human  vs.  Machine 


-  Analysis  of  the  Social  World 

Subject  Matter  Experts  (SMEs) 

Computer  Simulation 

Can  understand  human  contexts 

Limited  and  forced  understanding 

Can  recognize  new  situations 

Newness  (emergence)  not  well  developed 

Hard  to  get  statistical  significance  (exception: 
Massive  Multiplayer  Online  Gaming) 

Easy  to  repeat 

Human  variance  requires  more  repetitions 

Can  hold  all  else  the  same 

Individuals  stove-piped 

Scalable  and  crosscutting:  incorporates 
knowledge  from  many  disciplines 

Can  not  connect  micro  to  macro 

Can  compute  micro-macro  complexity 

SCIENTIFIC  INTEGRITY 


UNCLASSIFIED 


UNDERSTANDING 

uwSw  Synergies  between  Human  and  Machine  in  Oz 


Oz  supports  achieving  statistically  significant  patterns 


-  Allows  branching  and  keeps  track  of  the  branches 

-  Keeps  track  of  hierarchical  categorizations  of  moves  in  an  “ontology” 

•  Enables  post-game  statistical  analysis  and  data-mining 

-  Streamlines  the  move  input  and  adjudication  process 

•  Players  quickly  select  from  available  moves  in  a  menu 

•  Computer  models  suggest  adjudications  that  humans  may  check 

•  Rapid  entry  of  ontologies,  rules,  models,  and  data 

•  Human  resources  may  be  applied  to  more  repetitions  of  the  game 


•  Oz  does  not  limit  human  creativity 

-  Free  moves  are  allowed  in  the  war  game 

•  Players  may  suggest  a  new  categories 

•  Text  descriptions  are  stored 

•  In  extended  games,  computer  modelers  have  time  to  incorporate  new 
moves  into  their  models 

•  New  moves  are  easily  expressed  in  ontology  and  rules 

-  Human  adjudicators  have  the  final  say  over  model  suggested  results 
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Statistics  through  Branching 


For  example,  every  time  a 
particular  action  is  done,  or  a 
particular  player  makes  a 
move,  give  it  to  another  player 

-  For  pair-wise  comparison,  or 
random  block  design 
experiments 

•  Fewer  repetitions  needed 

-  “Holds  all  else  the  same”  by 
giving  the  same  history  up  to  the 
branching  point 

Done  behind  the  scenes 

-  Players  only  see  history  that 
they  should  see 

-  Perception  is  preserved 

-  Oz  file  sent  through  email 

Necessary  part  of  the 
Scientific  Method 

-  Done  in  the  United  Kingdom  and 
the  Army  War  College 
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Statistics  through  Ontology  Technology 


Ontology:  A  way  to  categorize  data 
into  general  and  specific  categories 

•  Intuitive  interface  for  input  through 
Protege  open  source  software 

Facilitates  significant  level  of 
aggregation  for  Statistics  and 
Data  Mining 

•  There  might  not  be  enough  data  on 
specific  terrorist  acts,  but  it  may  be 
significant  on  a  general  level 

•  Provides  gradient  for  data  mining 
techniques  ( like  MPICE,  CAST, 
ACTOR,  FORESITE) 


owl: 
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Finding  Patterns  in  Creative  Actions 


Q.  How  can  we  use  statistics  if 
Irregular  Warfare  Analysis  is 
Wicked? 

-  Doesn’t  human  creativity 
make  actions  unique? 

A.  We  aren’t  studying 

uniqueness,  we  are  studying 
patterns 


-  Unique  actions  still  fall  into 
types 

-  Statistics  measure 
coerciveness  of  action 

•  Defined  by  a  lack  of 
variance  in  response 

-  Medical  statistics  deal  with 
similar  levels  of  variation 
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The  Game  World  vs .  the  Real  World 


Q.  But  we  aren’t  using  real  data! 


A.  We  are  finding  patterns  in  our 
best  SME  and  model 
estimates 

•  Strategic  role-playing  helps 
players  to  “be  there” 

•  Statistical  comparisons  with 
real  data  can  eliminate 
“game  bias” 

Statistics  tease  out  the  effects 
due  to  the  game  itself  from 
the  effects  due  to  the 
idiosyncrasies  of  the  players 


Military  Operations  Research  Society 


i 

UNCLASSIFIED 


9 


SCIENTIFIC  INTEGRITY 
UNDERSTANDING 
TECHNOLOGY 


UNCLASSIFIED 


The  Model  Composition  Problem 


Q.  Isn’t  the  social  computation  in 
your  automated  adjudication 
another  wicked  problem? 

•  What  do  you  do  with  many 
perspectives? 

A.  Yes,  we  are  forced  to  compose 
social  simulations 

•  One  simulation  can’t  hold 
the  entire  social  world 

•  Each  social  scenario  is  a 
unique  combination 

•  Impractical  to  simulate  from 
scratch 

•  Needed  for  quick- 
turnaround  analysis 

•  Since  social  scientists 
disagree,  all  perspectives  of 
every  discipline  need  to  be 
tested 

...  and  we  are  applying  advanced 
technologies  to  the  problem 
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Composition  through  Ontology  Technology 


-  Ensures  Multi-Resolutional  models  can  speak  to  each 
other 


TECHNOLOGY 


•  Makes  a  mapping  between  simulations  possible 

-  An  action  at  a  lower  level  for  a  lower  resolutional  model  is 
automatically  mapped  to  a  higher  level  for  a  higher  resolutional  model 

-  Hub  and  Spoke  scheme  is  used 

•  Integrates  simulations  through  the  MVC  (model  view  controller) 
software  engineering  design  pattern 

-  Multi  resolutional  software  agrees  to  a  data  model,  and  consistency 
with  that  agreement  is  enforced 

-  Data  Model  is  not  buried  in  the  control  logic  of  the  simulation 


-  Enables  consistent  integration  with  data  in  databases, 
of  different  ontologies 


-  Facilitates  appropriate  levels  of  description  for  rules 


A  deep  ontology  allows  a  rule  to  be  general  or  specifi 
appropriate 
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Problem:  Consensus  Among  Social  Models 


UNCLASSIFIED 


•  r?The  social  disciplines  are  different  views  at  the  same  phenomena 

-  Overlap:  the  same  or  highly  correlated  events  are  covered  in  two  or 
more  simulations 

-  Conflicts  typically  occur  in  areas  of  overlap 

•  In  Oz,  models  may  be  synchronized  at  areas  of  overlap 

-  Many  conflict  resolution/synchronization  schemes  may  be  used 

•  Human  adjudication 

•  Weighted  voting  schemes 

-  Weeds  out  bugs  in  replicated  models 

•  Constraint  satisfaction 

•  Coevolution 


Module  1 


Module  3 


Module  2 


Module  4 


Social  models  overlap,  as  on  the  left,  instead  of  fitting  neatly  togeflie 
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Model  Consensus  through  Rule-Based  Systems 


•  in  Oz,  the  social  literature  itself  helps  achieve  consensus 

-  Both  types  of  social  literature  are  used 

•  Social  theory/causal  models  drive  simulation  modules 

•  Correlative  studies  designate  weighted  areas  of  overlap 

•  Correlative  rules  automate  integration  and  validation 

•  Models  and  model  combinations  that  best  fit  patterns  in  data  are  best 

-  We  can  not  expect  models  to  predict  events,  but  we  can  expect  them  to  match 
patterns 

•  Fuzzy  rulesets  model  correlative  studies 

-  Exactly  matches  the  data  of  correlative  studies 

•  Weight  of  rule  taken  from  correlation  coefficient 

-  Robust  with  respect  to  contradictory  data 

-  Fuzzy  Cognitive  Maps  implement  constraint  satisfaction  conflict  resolution 


TECHNOLOGY 
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Data  Aggregation  with  Fuzzy  Rules 


Combines  real-valued  model 
results  into  PMESII 
adjudications 

-  Correlative  Data  are  Social 
Indicators 

Scalar:  Can  determine 
degrees  of  change 

Intuitive  interface  for  input  with 
verbal  descriptions  of 
phenomena 

-  Open  source  JFuzzyLogic 

A  rule  from  PITF  correlative 
data: 

-  If  a  state’s  factionalism  is  high, 
and  its  democracy  is  partial,  then 
its  stability  is  low 

-  Calibrated  to  data 


-Jo \*\ 


tip 


0.0  2.5  5.0  7.5  10.0  12.5  15.0  17.5  20.0  22.5  25.0  27.5  30.0 


| —  cheap  —  generous  average  | 


^iQjxJ 


0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0  6.5  7.0  7.5  8.0  8.5  9.0 


—  rancid  — delicious 


NffBIIMH' 


^JnjxJ 


tip  :  11.701 603788948 043(CenterOfGravity) 


| — poor  —  excellent  good| 


If  the  food  is  delicious  and  the  service  is  excellent, 
then  the  tip  is  generous 


Military  Operations  Research  Society 


i 

UNCLASSIFIED 


14 


UNCLASSIFIED 


Automation  of  Wargame 


After  automated  adjudication,  consensus  is  exported  to  models 
for  them  to  restart  from 


•  Humans  may  also  change  fuzzy  rule  adjudications 

•  Human  modification  may  be  switched  off  for  automation 

-  Model-Game-Model  Process 


•  Al  in  game  may  generate  legal  moves  and  play  them 

•  Instead  of  taking  every  possible  move,  as  in  Data  Farming,  takes 
moves  according  to  strategy,  and  in  order  to  win,  as  in  Strategic 
Data  Farming 

-  If  a  computer  plays  COMPOEX  or  PSOM  better  than  people 
do,  its  better  to  automate 

•  Enough  runs  to  explore  space  of  possibilities 

•  Talk  over  the  meaning  of  moves  in  chess  never  won  the  game 
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The  Oz  User  Interface 


Q.  Isn’t  it  hard  to  both  smooth 
the  process  and  let  players  be 
creative  at  the  same  time? 


A.  Lets  look  at  the  interface... 


-  There  are  two  forms 

•  The  Move  Form 

•  The  PmESII 
Adjudication  Form 
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The  Move  Form 


The  Main  Page  is  the  Move  Form,  containing  information  on  individual 
moves 


•  Players  enter  overall  strategies  from  the  menu 

•  Players  enter  free  text  moves 

•  White  cell  can  enter  free  text  “screening”  adjudication 

•  Moves  are  categorized  so  they  may  be  entered  into  models,  rules,  and 
stored  for  statistical  analysis 

-  Players  enter  Actor,  Resource,  Time,  Location,  Target,  Intended 
Effects,  and  Strategy 

-  White  cell  enters  visibility  of  the  action 

-  If  there  is  no  appropriate  category,  a  new  one  may  be  entered  into  the 
existing  ontology 

•  Historical  forms  are  filtered  according  to  what  is  visible  to  the  player 

-  A  Timeline  shows  historical  moves 


-  They  may  be  further  filtered  based  on  the  categorizations 
Game  may  be  branched  on  particular  moves  for  comparison 
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The  Move  Form 
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describe  Strategies  and  Enter  New  Categories  Through  Control  Menu 
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Navigate  History  with  Timeline  and  Filter 


New  Mt 


View  |  Control  Help 
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Filter  Move  Histoi 
Switch  Player 
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Categorization  Buttons  Bring  Up  Categorization  Tree 


File  Edit  View  Control  Help 

New  Move  Validation 
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Game  Branched  and  Moves  Exported  to  Models  through  File  Menu 


-ifli  >1 


File  ]  Edit  View  Control  Help 

New ... 

Open  Game ... 

Save  Game ... 

Print  Branch ... 

Import  Ontology ... 

Export  Model  Data:  Moves  v 
Exit 


rN®  Moves  Present- 


Properties  f  Screening 

Move 
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Save  In: 
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Checklists  Help  Players  Keep  Track  of  the  Process 


New  Move 

Player  Checklist 

Move  Checklist 

User  Manual  \ 

ition 

Create  Empty 

ick 

O  Identify  Yourself  (View  Menu :  Switch  Player ... ) 

O  Import  Ontologies  (File  Menu :  Import  Ontology ...) 

O  New  or  Open  Branch  To  Be  Adjudicated  (File  Menu 

Need  Time 

Jan2014  \ 

Player  Checklist 


O  Create  a  Blank  Move  Form  (File  Menu :  New :  Move ) 

O  Describe  Overall  Strategy  (Control  Menu :  Describe  Strategy ...) 
O  Enter  Text  and  Classifications  of  Move  (Move  Panel) 

O  Save  Branch  (File  Menu:  Save  Branch ...) 


Reset 


T 


No  Moves  Present- 

f  Properties  |  Screening  | 


Move  Adjudication  Checklist 


O  Identify  Yourself  (View  Menu :  Switch  Player ... ) 


Description 


O  Import  Ontologies  (File  Menu :  Import  Ontology ...) 

O  Open  Branch  To  Be  Adjudicated  (File  Menu :  Open  Branch ...) 

O  Export  Data  for  Model  Adjudication  (File  Menu:  Export  Model  Data:  Moves ...) 

O  Adjudicate  Individual  Moves,  in  Text  and  Categorizations  (Adjudication  Panel) 

O  If  necessaiy,  Print  Branch  for  separate  SME  Adjudication  (File  Menu:  Print  Branch ...) 
O  Save  Branch  (File  Menu:  Save  Branch ...) 
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White  Cell  Screens  Moves 


File  Edit  View  Control  Help 
New  Move 


Create 


Validation 

Empty 

■W 

Check 

Report 

□  Automatic 

Filter 


□  On/Off  Setup 


Timeline  By  Month 


Refresh 


Setup 


Need  Time 


Jan2014 


Feb2014 


Mar2014 


Apr2014 


May2014 


Jun2014 


Negotiate  Favorable 
Trade  Agreements 


iFinancial 


Cyber  Strike 


Branch 
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Strategy 


Move:  3  of  3  (  Financial  ) 

j'  Properties  j  Screening  | 


Screening  Description 


Screening 


Ftesult 

Screeners 

Visibility 

Blue  Cell 

Properties  Summary 


time:  WeekOS 
player:  Blue  Cell 
location: 
effect: 
target: 
resource: 
action:  Financial 
strategy: 

actor:  U  S  Air  Force 
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The  PM  ESI  I  Adjudication  Form 


•  Accessible  from  the  Control  Menu,  so  that  Historical  PMESII 
Adjudications  may  be  examined 

-  History  is  navigated  using  back  and  forward  buttons 

•  Adjudicators  import  model  results,  rule  sets,  and  answer 
questions  that  aren’t  covered  by  models  and  rule  sets 

•  PMESII  adjudications  are  for  a  particular  Time,  Location,  and 
Actor 

•  Rule  sets  based  on  Social  Indicators  roll  up  the  results  to 
PMESII  values 

•  Adjudicators  may  modify  both  specific  indicator  results  and 
general  PMESII  results 

•  Adjudicators  may  export  final  adjudications  back  to  models 
so  that  they  all  restart  from  the  consensus  state 


TECHNOLOGY 
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The  PMESII  Adjudication  Form 


File  Edit  View  Help 


Imports 

Ftiilesets: 

Models: 


Group 


Scope 

Time 

Location 

Actor 

Adjudicator 

Automate 


Questions  and  Answers 


Question 


Answer 


Comment 


Data  Resources 

Method 

State 

Affected  Method 

Affected  State 

Value 

Final  Value 

Reason 

Metric 

Automated  Value 

Automated  Effectiveness 

Final  Value 

Reason 

Defection  Rates 

A. 

Financing 

= 

10 

Recruitment 

T 

C 
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Population 


PMESII  Dimension 

Automated  Adjudication 

Automated  PMESII  Result 

Final  PMESII  Result 

Reason 

Political 

Military 

Economic 

Social 

Infrastructure 

Information 
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A  Checklist  Guides  Adjudication  as  on  Move  Form 


Jn|x| 


W  Help 


Imports 


PMESII  Adjudication  Checklist 


Riilesets: 


Models: 


This  Presentation 


Data  Resources 


Method 

State 

Affect 

Group 


Metric 

Automated  Vali 

Defection  Rates 

Financing 

10 

Recruitment 

Population 


PMESII  Dimension 

Automated  Adiudi 

Political 

Military 

Economic 

Social 

Infrastructure 

Information 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


Identify  Yourself  ( Menu  Form:  View  Menu :  Switch  Player ... ) 

Import  Ontologies  (Menu  Form:  File  Menu :  Import  Ontology ...) 

Open  Branch  To  Be  Adjudicated  ( Menu  Form:  File  Menu :  Open  Branch ...) 

Open  PMESII  Adjudicator  (This  Form)  (Menu  Form:  Control  Menu:  PMESII  Adjudication) 

Create  a  Blank  PMESII  Adjudicator  Form  (File  Menu:  New  Adjudication) 

Enter  Time,  Location,  and  Actor  to  be  Adjudicated  (Scope  Panel) 

Import  Model  Data  (File  Menu:  Import  Model  Data ...) 

Review  Model  Data  and  if  necessary,  modify  indicator  values  and  give  a  reason.  (Data  Ftesources  Panel) 

Revisit  Move  Adjudications  and  if  necessary,  modify  move  adjudications.  (Move  Form:  Adjudication  Panel) 
Import  PMESII  Ftuleset  (File  Menu:  Import  PMESII  Ftuleset...) 

Answer  Questions  (Questions  and  Answers  Panel) 

Run  PMESII  Ftuleset  (Automate  Button) 

Review  Group  Results,  and  if  necessary,  Modify  Final  Group  Result  and  give  Fteason  (Group  Panel) 

Review  Population  Results,  and  if  necessary,  Modify  Final  Population  Result  and  give  Reason  (Population  Panel) 
Save  Adjudications  (File  Menu:  Save  Adjudications ...) 

Export  Data  so  Models  may  conform  to  consensus  state  (File  Menu:  Export  Model  Data:  Final  Adjudication...) 
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Results  Are  Rolled  Up  with  PMESII  Ruleset,  Edited  and  Exported  Back 

to  Models 


File  |  Edit  View  Help 


^jnjxj 


New  Adjudication 
Save  Adjudications ... 

Print  Adjudications ... 

Import  Model  Data ... 

Import  PMESII  Ruleset ... 

Export  Model  Data:  Final  Adjudication ... 


Location 

Actor 

Adjudicator 

Questions  and  Answers 


Models: 


Question 


Answer 


Export  Data, 


Resources 


Group 


Automate 


Comment 


Method 

State 

Affected  Method 

Affected  State 

Value 

Final  Value 

Reason^^ 

Metric 

Automated  Value 

Automated  Effectiveness 

Final  Value 
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A. 

Defection  Rates 

Financing 
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Input 

Fields 
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PMESII  Dimension 

Automated  Adjudication 

Automated  PMESII  Result 
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Rea^m 
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Information 

OK 

Cancel 

Military  Operations  Research  Society 


i 

UNCLASSIFIED 


28 


SCIENTIFIC  INTEGRITY 
UNDERSTANDING 
TECHNOLOGY 


UNCLASSIFIED 


Summary 


Wargaming  is  a  useful  tool  for  analyzing  Wicked 
Problems 


•  Technology  can  assist  wargaming  adjudication 

•  Oz  is  unique  because  it 

-  Integrates  data,  rules,  and  models  in  the  wargaming 
environment 


-  Incorporates  Social  Science  theory 

-  Integrates  the  results  of  multiple,  multi-resolution  models 

-  Preserves  unique  perspective  of  each  wargame 
participant 

-  Allows  branching  of  the  wargame 

-  Preserves  a  record  of  the  wargame  for  subsequent 
statistical  analysis 


Can  be  easily  adapted  to  a  variety  of  warga 
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Timeline  Moves  Color  Coded  By  Category 


File  Edit  View  Control  Help 


New  Move 

Validation 
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Report 

□  Automatic 
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Setup 


Timeline  ley  Month  W 


Negotiate  Favorable 
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iFinancial 


|Cyber  Strike  | 


May2014 
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Fill  Color  Property  for  Move 
Action  j  ▼ 


Timeline  Display  Properties 

Property  to  Show  for  Move 


Branch 
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Choose  Display  Color  For  Property 


Move:  3  of  3  (  Financial  )- 

Properties  [  Screening  j 


screening 

Result 

Screeners 

Visibility 

Blue  Cell 

Properties  Summary 

time:  WeekOS 
player:  Blue  Cell 
location: 
effect: 
target: 
resource: 
action:  Financial 
strategy: 

actor:  U  S  Air  Force 
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Categorization  Trees  are  Imported  from  Protege 

Ontology  Software 
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Moves  may  be  Reused  and  Reordered  through  the  Edit  Menu  or  the  Timeline 


Edit  View  Control  Help 


Validation 
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T 
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Report 

□  Automatic 
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Setup 


Timeline 


Refresh  Setup 


Apr2014 


May2014 


Select  and  set  time  below 


Branch 
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History  is  filtered  by  Player  Visibility  and  Categories  as  on  the  Move 

Form 


^JS|X] 


Scope 


View  Help 


Go  To  Time  Period ... 

Filter  Adjudication  History ... 
Switch  Player ... 


Location 
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Adjudicator 

Questions  and  Answers 


Question 


Automate 
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Comment 
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Example  Game  Turn  Cycle 


•First  two  turns  will  be  4  weeks  IZ 
•Last  three  turns  will  be  3  weeks 


Review/Edit  Automatic  PMESII  Results 
(Calculated  using  fuzzy  rules  with  model, 
BOGSAT  and  conditional/generic  rule 
data) 

Answer  Indicator 
Questions  Utilizing  the 
Model  Data  and  Action  '  Create 

Results  V  ^urn 

Outbriefs 


Teams  Enter  Turn  Data 
into  Oz 


Determine  Action  Outcomes 
(BOGSATs  and  SMEs  only) 
AND 

Model  Adjudication 


Review/Edit  Model  ***. 
Impacts  on  Action 
Outcomes 


1  5  business  days  < 

7  calendar  days 

[  10  business  days  \ 

14  calendar  days 

5  business  dlys  j 

7  calendar  days  ; 

• 

3  business  days 

3  calendar  days 

y 

5  business  days 

*♦.  7  calendar  days 

*.  ♦ 

• 

7  business  days  / 

11  calendar  days 

* 

Export  Turn 
Data  to  Models 


Action 

Adjudication 


Import  Model 
Data 

Send  Turn 

Adjudication  Results 
to  Modelers 


PMESII' 

Adjudication 
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